Aiming at the problem that the deraining methods based on tensor product wavelet cannot capture high-frequency rain streaks in all directions, a Dual U-Former Network (DUFN) based on non-separable lifting wavelet was proposed. Firstly, the isotropic non-separable lifting wavelet was used to capture high-frequency rain streaks in all directions. In this way, compared with tensor product wavelets such as Haar wavelet, which can only capture high-frequency rain streaks in three directions, DUFN was able to obtain more comprehensive rain streak information. Secondly, two U-Nets composed of Transformer Blocks (TBs) were connected in series at various scales, so that the semantic features of the shallow decoder were transferred to the deep stage, and the rain streaks were removed more thoroughly. At the same time, the scale-guide encoder was used to guide the coding stage by using the information of various scales in the shallow layer, and Gated Fusion Module (GFM) based on CBAM (Convolutional Block Attention Module) was used to make the fusion process put more focus on the rain area. Experimental results on Rain200H, Rain200L, Rain1200 and Rain12 synthetic datasets show that the Structure SIMilarity (SSIM) of DUFN is improved by 0.009 7 on average compared to that of the advanced method SPDNet (Structure-Preserving Deraining Network). And on Rain200H, Rain200L and Rain12 synthetic datasets, the Peak Signal-to-Noise Ratio (PSNR) of DUFN is improved by 0.657 dB averagely. On real-world dataset SPA-Data, PSNR and SSIM of DUFN are improved by 0.976 dB and 0.003 1 respectively compared with those of the advanced method ECNetLL (Embedding Consistency Network+Layered Long short-term memory). The above verifies that DUFN can improve the rain removal performance by enhancing the ability to capture high-frequency information.